United States Patent [i9] 

Miller et al. 



US006151696A 
[ii] Patent Number: 6,151,696 
[45] Date of Patent: *Nov. 21, 2000 



[54] DATA TRANSMISSION 

[75] Inventors: C. Kenneth Miller, Concord; Kary 
Robertson, Newburyport, both of 
Mass.; Kenneth Cates, Salem, N.H.; 
Marc White, Wayland, Mass. 

[73] Assignee: StarBurst Communications 
Corporation, Concord, Mass. 

[ * ] Notice: This patent is subject to a terminal dis- 
claimer. 

[21] Appl. No.: 09/012,386 
[22] Filed: Jan. 23, 1998 

Related U.S. Application Data 

[63] Continuation of application No. 08/585,948, Jan. 16, 1996, 
Pat. No. 5,727,002, which is a continuation-in-part of appli- 
cation No. 08/375,493, Jan. 19, 1995, Pat, No. 5,553,083. 

[51] Int. CI. 7 H04L 1/16 

[52] U.S. CI 714/748 

[58] Field of Search 714/748, 749 

[56] References Cited 

U.S. PATENT DOCUMENTS 

4,493,021 1/1985 Agrawal et al 364/200 

4,554,656 11/1985 Budrikis et al 370/85 

4,759,015 7/1988 Takai et al 370/86 

4,888,767 12/1989 Furuya et al 370/95.2 

4,914,654 4/1990 Matsuda et al 370/94.1 

4,937,819 6/1990 King 370/95.3 

5,296,936 3/1994 Pittas et al 358/407 

5,297,143 3/1994 Fridrich et al 370/85.3 

5,553,083 9/1996 Miller 371/32 

5,572,678 11/1996 Homma et al 395/200.12 

5,619,689 4/1997 Kelly et al 395/617 

5,727,002 3/1998 Miller et al 371/32 

FOREIGN PATENT DOCUMENTS 

0 303 830A2 2/1989 European Pat. Off. . 

0 437 072 Al 7/1991 European Pat. Off. . 

0 552 794 A2 7/1993 European Pat. Off. . 

0 654 921 Al 9/1994 European Pat. Off. . 



91/13503 2/1991 WIPO . 
95/26088 10/1995 WIPO. 

OTHER PUBLICATIONS , 

Towsley, "An Analysis of a Point-to-MulUpoint.-Crlarinel 
Using a Go-Back-N Error Control Protocol", IEEE Trans- 
actions on Communications, vol. COM-33, No. 3, Mar. 
1985, pp. 282-285. 

(List continued on next page.) 

Primary Examiner — Stephen M. Baker 

Attorney, Agent, or Firm — Testa, Hurwitz & Thibeault, LLP 



[57] 



ABSTRACT 



A data transmission method quickly and reliably transfers 
data (e.g., a co mputer file Afroi n a source to- recipients. While 
the"frames are being transmitt ^jiegative acknowledgments 
frorrTf^cyient s^are receive d JkyJhe^scai-Tce . These acknowl- 
edgments indicate which frames require retransmission. 
After all frames have been transmitted out, a retransmission 
is performed by the source for only those frames which the 
acknowledgments indicate require retransmission. Addi- 
tional retransmissions can occur. This multi-pass data trans- 
fer technique requires only negative acknowledgements to 
be sent by the recipients. Features include the ability to set 
the transmission rate and to define multicast groups. Also, it 
is possib le to determine the capacity of links of unknown 
capacity using a "multicast network prob"e"~feature of the 
invention, and to determine the frame error rates of known- 
capacity links by utilizing the same feature. A "multicast 
ping" feature of the invention can be used to determine the 
connectivity between a source and members of a multicast 
group. "Speed groups" can be set up after determining link 
capacities, or if they are already known, whereby the recipi- 
ents connected to the source by the fastest links receive all 
of the data while slower-link recipients receive only a 
portion of the data, on the first pass. The number of 
recipients which can receive the data from the source can be 
greatly increased by using a "negative acknowledgement 
collection" scheme whereby "replication points" (preferably 
routers) collect individual negative acknowledgements and 
forward them as a unit to the next level. 

23 Claims, 9 Drawing Sheets 
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DATA TRANSMISSION the network. CDPD wireless networks recommend applica- 

tions operate over UDP (the connectionless transport layer) 
CROSS-REFERENCE TO RELATED CASES only, and thus TFTP is the file transfer protocol of choice for 

CDPD 

This is a continuation of U.S. patent application Ser. No. . ' 

08/585,948, filed on Jan. 16, 1996, now U.S. Pat. No. S TFTP breaks files up into packets having 512 bytes of data 
5,727,002, which is a continuation-in-part of U.S. patent eac ?> ? nd " th , en ^ ea =t daU P acket one at a time - ^ 
application Ser. No. 08/375,493, filed on Jan. 19, 1995, now eac . n data P ack ? 15 « n > TFTP causes the sendin S node }° 
U.S. Pat. No. 5,553,083, which was reexamined as reex- ™" for an acknowledgment from the receiving node(s) 
amination control number 90/005,028. For reexamination in bef ° re *J^^ V * a ^ 0Wed ,0 ^ nd . th6 next , data 
control number 90/005,028, a Notice of Intent to Issue 10 P acke »- ™ 15 described, for example, in a book by 
Reexamination Certificate was mailed on Jan. 18, 2000. Douglas E. Comer (Internetworking with TCP/IP, Volume I, 

Principles, Protocols, and Architecture, Second Edition, 
FIELD OF THE INVENTION Prentice Hall, 1991, Chapter 23, pages 377-390). 

_ . . . •• j While acknowledgment is a part of TFTP, the acknowl- 

Tms invention relates to data transmission, and more is e dgment scheme used in TFTP becomes very inefficient as 
particularly, to fast and reliable multicast transmissions of network delay becomes signiflcant and/or is different for two 
files from a server to clients. of more of (he receiving nodes Lflce XFTPj some other 

BACKGROUND OF THE INVENTION known data transfer mechanisms require packet-by-packet 

acknowledgment, and thus these other mechanisms also are 

Computer networks, such as wide area networks (WANs), 20 relatively slow at transferring the entire amount of data, 
can provide unicast, multicast, and broadcast services to 

allow communication between network participants such as SUMMARY OF THE INVENTION 

a server node and one or more client nodes. Multicast frame uism objec , of ^ present imen&m to provide botn fast 

relay is a service used to communicate over a computer ^ reJiable transmission of flles from a server t0 one or 

network. Multicast IP technology is another service used to * more cIien(s over a ^^^^ link . ^ file ^vster 

communicate over a computer network. Broadcast frame pre ferably is a multicast transmission to clients. In general, 

relay is a service used to communicate over a satellite fi , 6 tasa&t accordiag to the mv6ntion ^ not suffet any 

network. The term "broadcast refers to a server node reduction in speed, reliability, or efficiency in the face of link 

sending information to all of the client nodes connected to dc l a y, even if that delay is significant and/or different for two 

the network. The term multicast refers to a server node or more of ^ receiving clients . ^ provides ^ 

sending information to a subset of all of the client nodes ideal mechanism for distributing computer software files 

connected to the network. Broadcast and multicast are electronically 

network capabilities which are relatively new over WANs. _ • „• i- i . • L i n_ * *i_ 

r J The communications link, which couples the server to the 

Some information providers desire to deliver information 35 clieQts and allows communication therebetween, can be a 

electronically by broadcasting or multicasting the informa- compu ter network (e.g., a LAN, a WAN, the Internet), a 

tion from a server node at a central location to one or more wireless network (e.g., a packet cellular data network such 

client nodes at remote customer locations via a computer as CD p D ) ) ^ combination of these types of communi- 

network to which the server and the clients are coupled. cation mediums, or some other communication medium 

Because broadcast and multicast network services do not 4Q such aSj for example, a satellite network which generally is 

provide for acknowledgment of the delivered information at a high-speed high-delay network. 

all, these services can be unreliable. Such unreliability [fl accordance with me mvention> the clients send only 

generally is undesirable and unacceptable to information 1 * * . u i + *u *u 

■j negative acknowledgments back to the server as the server 

P is sending the data files. The communication is continuous. 

A common protocol suite in use in computer networks is 45 ^ ^ the servcr does not stop sen ding the data to wait for 
TCP/IP, which is the protocol used in the Internet. TCP me negative acknowledgments from the clients, but instead 
stands for Transmission Control Protocol, and IP stands for thc serycr receivcs me clients > neg ative acknowledgments as 
Internet Protocol. Two file transfer protocols are available in me server fc transmitting the data. The clients' negative 
association with TCP/IP: (0 File Transfer Protocol (FTP) acknowledgments indicate to the server which particular 
which runs as an appUcation on top of TCP and (u) Trivial 50 packets need to be resent. A packet may need to be resent 
File Transfer Protocol (TFTP) which runs on top of UDP. because> for example , it was either not received or received 
UDP stands for User Datagram Protocol. Both TCP and m error by one or more of the clients . M t^ the server has 
UDP are transport protocols which are responsible for sent the entire amount of data (e.g., the entire file) over the 
end-to-end delivery of information across an internetwork, Hnk to me clients? the server per f orms a second round 0 f 
i.e., a network o f netw orks. 55 transmissions in which it only resends the particular packets 

Both FTP and TFTP support point-to-point (i.e., unicast) indicated by the clients as requiring retransmission. During 
file transfers only. FTP depends on TCP for reliable delivery, this second round, clients still only send negative acknowl- 
as TCP is a connection-oriented acknowledged transport edgements (i.e., indications of packets not received at all or 
protocol. TFTP provides its own acknowledgments for not received correctly). The process can then continue with 
reliability, as it runs on top of UDP which is a connectionless 60 as many additional rounds of retransmissions as is required 
transport service that does not support acknowledgment. S o that each of the clients correctly receives all of the 

Connection-oriented protocols such as TCP require setup packets. Alternatively, the retransmission rounds can be 
and tear-down of virtual circuit connections. Because of repeated a predetermined number of times, which number 
their relatively high overhead, TCP and similar protocols are can be modified (i.e., the number is configurable). Each 
undesirable in networks with inherently poor connections 65 subsequent round typically involves the transmission of 
such as Cellular Digital Packet Data (CDPD) networks. fewer packets than the previous round, as only previous 
CDPD utilizes TCP/IP as the primary protocol suite used in packets in error are resent. 
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This scheme quickly and reliably transfers data from a _ .group. "Speed groups" can be set up after determining link 

server to one or more clients. It is quick because the server capacities, or if they are already known, whereby the recipi- 

is allowed to transfer the entire file without stopping at ents connected to the source by the fastest links receive all 

packet boundaries to wait for negative acknowledgments 0 f the data while slower-link recipients receive only a 

from the clients for the packet just sent. That is, data transfer 5 portion of the data, on the first pass. The number of 

is not directly tied to negative acknowledgments in that each recipients which can receive the data from the source can be 

round of data transfer continues regardless of any particular greatly increased (e.g., by a factor of 1000 or more) by using 

client's reception problems and/or regardless of any link a « ncgat j ve acknowledgement collection" scheme whereby 

delay issues (e.g., a difference in the time it takes a packet "replication points", preferably routers, collect individual 

to travel from the server to a certain client and the time it 10 neg ative acknowledgements and forward them as a unit to 

takes a packet to travel from the server to another different tne next ^ eve j 

client). Also, each subsequent round of transmission only „ ^ noted ^ ^ ^ . ket , aQd Ifame , 

involves the sending of packets which were not received or are ^ mtercn bl he H reiD t0 identi | ^ Mme ^ 

received in error during the previous round and therefore n a ^ of ^ > f infonnation ^h may have a 

the server generally does not ever need to send the entire file -.c „ „« . f • , , _ *u^*e „,u;„i, .v 

& _* . , , . . A A . . .j source and destination address as part thereof and which is 

more than once. It is reliable because it strives to provide & ^ across the link 

each client with every packet, and the reception problems of . ' 

any individual client generally does not affect the other ™* Agoing and other objects, aspects, features, and 

clients 1 reception speed and accuracy. advantages of the invention will become more apparent from 

Data transfer according to the invention does not require ao me followin e Ascription and from the claims, 

or expect positive acknowledgements from any of the cli- BRIEF DESCRIPTION OF THE DRAWINGS 
ents. A positive acknowledgement is implicit if a negative 

acknowledgement is not received back at the server. In the drawings, like reference characters generally refer 

Moreover, in accordance with the invention, a plurality of to the same parts throughout the different views. Also, the 

negative acknowledgements preferably are collected and 25 drawings are not necessarily to scale, emphasis instead 

sent back to the server as a "multiple selective reject generally being placed upon illustrating the principles of the 

negative acknowledgement." Typically, more than one of invention. 

these multiple selective reject negative acknowledgements FIG x is a fl owcn art of data transmission operations 

are sent back to the server during, for example, the first according to the invention. 

round of transmissions from the server to the clients One 30 FIG . 2 ^ a di of a h ical configuration which 

multiple selective reject negative acknowledgement can a to communicate ^ one or morc clients< 

represent hundreds of individual negative acknowledge- . .«.,.* # 

ments. The use of these collections of negative acknowl- FIG - 3 [ s a diagram showing the location of an embodi- 

edgements can greatly reduce traffic over the link and free up ment of ^ invention m relation to the TCP/IP protocol 

bandwidth on the link for the transfer of data from the server 35 stack - 

to the clients and for other uses. With the invention, the FIG. 4 is a diagram of a "first pass" block and frame 

server and the link generally do not get choked with indi- transmission and acknowledgment process according to the 

vidual negative acknowledgements all coming back at the invention. 

same time or within a short window of time. This reduction FIG. 5 is a simplified block diagram of a server in which 

in the number of individual acknowledgements being sent 40 at least a portion of the present invention can be embodied, 

over the link to the server also results in the benefit and FIG 6 ^ a fagrzm of a heterogeneous multicast network 

significant advantage of improved scalability. That is, with wiih mem bers of a multicast group connected by different 

the use of multiple selective reject negative capacity finks 

acknowledgements the number of clients to which . tflle can pjQ, 7 is a diagram illustrating an acknowledgement 

be sent increases due to the reduced acknowledgement 45 collectionfeaturea * ordin g to the invention which increases 

traffic coming back to the server scalability and allows millions of recipients to receive 

In a preferred embodiment of the uivention, the entire ic]d ^ reliab] data &Qm a sender 

amount of data to be transferred (e.g., a file) is separated into ,. 1x _ . ,„ ^ t 

a plurality of blocks, where each block includes a plurality . 8 15 * related to congestion/flow control 

of packets. The server completes a round when it finishes 50 USing a VamWe bl ° ck S1ZC method 

transmitting all blocks (e.g., the entire file). After a complete FIG. 9 is a diagram related to congestion/flow control 

block has been transmitted, the clients send their negative using a preferred status request method to solicit negative 

acknowledgments back to the server via a return unicast acknowledgements from clients before block boundaries, 

communications path. Block boundaries trigger the sending nF^PRipnnNr 

of negative acknowledgments by the clients. As the negative 55 

acknowledgments are coming into the server from the Referring to FIGS. 1 and 2, in accordance with the 

clients for block N, the server is transmitting block N+l (or invention, quick and reliable data transmission from a 

a subsequent block) out to the clients or the server has source or server 20 to one or more recipients or receivers or 

finished transmitting all of the blocks. clients 22 lr 22 2 , . . . , 22^ over a communications link 24 

The following features are provided according to the 60 comprises (step 10) transmitting the data (e.g., a file), which 

invention. There is the ability to set the transmission rate and is in the form of a plurality of frames, over the link 24 to one 

to define multicast groups. Also, it is possible to determine or more of the recipients 22 until the entire file (i.e., all of 

the capacity of links of unknown capacity using a "multicast the plurality of frames) have been transmitted over the link 

network probe" feature, and to determine the frame error 24. As the frames are being transmitted, frame negative 

rates of known-capacity links by utilizing the same feature. 65 acknowledgments from one or more of the recipients 22 are 

A "multicast ping" feature can be used to determine the received via the link 24 (step 10). If, after the entire file has 

connectivity between a source and members of a multicast been transmitted over the link 24, the negative acknowledg- 
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ments indicate that certain frames need to be retransmitted top of UDP. The invention also could operate at the appr- 
over the link 24 (step 12), only those certain frames are cation layer above the connectionless transport layer present 
retransmitted (step 14). As those certain frames are being in other protocol stacks such as IPX in the NetWare SPX/ 
retransmitted over the link 24, frame negative acknowledg- IPX protocol suite. UDP stands for User Datagram Protocol, 
ments from one more of the recipients 22 are received via 5 and it is the TCP/IP standard protocol that allows an appli- 
the link 24 (step 14). This process is then repeated as many cation program on one computer to send a datagram to an 
times as necessary until no more frames need to be application program on another computer. UDP uses the 
retransmitted, as indicated by steps 12, 14, and 16. In step Internet Protocol (IP) to deliver datagrams. UDP datagrams 
16, the server 20 determines whether "done" messages have m&x from Ip dat agrams in that UDP datagrams include a 
been received at the server 20 by all of the recipients 22. If 1Q tocol rf numbef which ^ ows ^ of the data . 
a recipient is "done, it means that reapient has received all to ^ ^ a muM k destinations (i . e appli . 
of the frames and has sent to the server 20 a done message ^ on the receiyin tcr UDP V datagrams 

to so indicate. Done recipients continue to send done . * • n • , . ^ 1 c *i_ j * u * * 

, .., r 4 . - « A also typically mclude a checksum for the data being sent, 

messages to the server until they see their name in a done j & 

list" which the server sends out as a notification to all "done" In S eneral > data transmission according to the invention 

recipients (i.e., those listed in the "done list") to stop sending 35 includes four aspects: IDLE, ANNOUNCE/ 

"done" messages to the server. After a predetermined period REGISTRATION, TRANSFER, and COMPLETION. In the 

of time or after a predetermined event, the server 20 sends IDLE state, there is no activity. When a collection of data 

a status request to all unresponsive recipients 22, i.e., (e.g., a file) is selected for transmission by the server 20, the 

recipients from which it has not received a "done" message ANNOUNCE/REGISTRATION phase is entered. During 

(step 18). The initial transfer of the entire file and each of the 20 any of the four phases, all files are available to an operator 

subsequent transmissions of error frames are generally at the server 20. 

referred to herein as a "round" or "pass". Announce/Registration 

In the first pass, the server 20 preferably multicasts the file In this phase (step 8 in FIG. 1), the server ANNOUNCES 

to a subset of all of the clients 22. At least two of the clients to the clients that a file is about to be transferred and 

22 typically have a different server-to-client frame transmis- 2 5 provides the parameters associated with the transfer of the 

sion delay associated therewith. Data transmission accord- fii e . The maximum duration of this phase is expressed in 

ing to the invention is unaffected by such delay differences minutes, and it is configurable. An ANNOUNCE message is 

even if significant and even if every client 22 has a different used to ^ up mu iticast groups, and Class D addresses are 

delay associated therewith. used ^ ^ assignment of multicast groups. 

The link 24 can be a computer network (e.g., a LAN, a 30 clients are obliged to register with the server that they 

WAN, the Internet), a wireless network (e.g., a cellular data received an ANNOUNCE message. When a client sees the 

network), some combination of these two types of commu- ANNOUNCE message, the client verifies that it is associ- 

nication mediums, or some other communication medium aled witn me gr011 p identified in the message. It is implicit 

such as, for example, a satellite network which typically are m lhc receiver able to pr0C ess the ANNOUNCE 

high-speed, high-delay. The plurality of frames transmitted 35 messa ge that the receiver has a correct server IP address and 

over the link 24 during the first round can together represent a mmtii port nlimben clients automatically respond to 

a computer data file being transferred from the server 20 to ANNOUNCE packets with REGISTRATION packets until 

one or more of the clients 22. mcy see ^ address in a registered client list in a subse- 

The server 20 and the clients 22 can be computers, such queat ANNOUNCE packet. The REGISTRATION packet 

as PCs or workstations, running any one of a variety of 40 acts as a positive acknowledgment to the server about the 

operating systems including DOS. Referring to FIG. 5, the client's participation. Once the server receives the client's 

server 20, regardless of what type of computer it is, typically REGISTRATION packet, the server adds the client to the 

includes a central processor 50, a main memory unit 52 for cn ent list in the next broadcast of the ANNOUNCE packet, 

storing programs and/or data, an input/output controller 54, The client list is maintained by the server. When the client 

a network interface 56, one or more input devices 58 such 45 receives an ANNOUNCE packet with the clients ID in the 

as a keyboard and a mouse, a display device 60, a fixed or c ii ent ]i st> registration for the client is complete. When all 

hard disk drive unit 62, a floppy disk drive unit 64, a tape expected receivers have responded to the ANNOUNCE 

drive unit 66, and a data bus 68 coupling these components message or the ANNOUNCE timeout has expired, which- 

to allow communication therebetween. Each of the client e ver comes first, actual transmission of the file will begin, 

computers 22 generally includes all or some of the compo- 50 This registration indicates that the client can participate in 

nents included in the server 20 of FIG. 5. the group, as it has the resources to handle the file about to 

In some embodiments, one or more computer programs be sent. To prevent unwanted participation, encryption key 
define the operational capabilities of the server 20 and the exchange can take place at group setup. Once file transfer 
clients 22. The programs can be loaded into the server 20 begins, ANNOUNCE packets cease to be sent, and the 
and the clients 22 via the hard drive 62, the floppy drive 64, 55 ANNOUNCE phase is over (step 9 in FIG. 1). 
and/or the tape drive 66. Alternatively, the programs can All the characteristics of the file transmission are trans- 
reside in a permanent memory portion (e.g., a ROM chip) of mitted in the ANNOUNCE packet. On receiving this 
the main memory 52. In some other embodiments, the server ANNOUNCE message, the client responds with a unicast 
20 and/or the clients 22 can include specially-designed, datagram to the server. The response indicates whether or 
dedicated, hard -wired electronic circuits which perform all 60 not the receiver has the facilities to receive the file. It also 
functions described herein without the need for instructions indicates, in the case of an aborted transmission, whether the 
from computer programs. The invention can be used, for client has enough context to resume the transmission (a 
example, to load quickly and reliably new revision levels of "restart" as indicated in FIG. 1). The duration of the 
the client software electronically from the server onto one or announce period in some instances should allow for an 
more of the clients. 65 operator at the server site to initiate a call to the client site 

Referring to FIG. 3, the invention preferably operates at indicating that the computer is either not available or does 

the application layer 30 of the TCP/IP protocol stack 32 on not have the facilities for the transfer. At the client site, the 
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corrections could be made either manually or, if so . configurable. This represents the basic transfer rate that may 

configured, under remote control from the server to free up be throttled back (i.e., decreased) based on performance. 

resource so it can participate in the transfer. The server continues sending the frames of the file until the 

At any point in time throughout the transmission, the complete file has been sent once into the network (i.e., until 

client may respond to this packet indicating that it aborted 5 Bloc^ through Biock 4 are sent). This is defined as the first 

the transmission from its end indicating the reason in the P 3 ^ or first round > 111(1 il takes ^ amount of time repre- 

message. If a transfer is broken off before completion, the sente( * in £ IG ;. 4 as Some clients may receive the 

invention is able to resume later without resending parts of com P iete fi u le < 1C > ^ t four t bl "*?\ after * e ^ 

the file already sent successfully ("restart" in FIG. 1). This PJ? s > 4 m which . case the V have ^ hcd r * cemQ g the file ; 

* ii i • ^ * f *, „u a- Chents receiving one or more frames in error, or not 

is an especially important and useful feature when sendmg 30 . . * n j- c 

i £i ^ i.- t.j . receiving one or more frames at all, require the resending of 

very large files To achieve this feature a client does not c ^ f of ^ file (i ^ er ? oneous i y „ receive S or 

discard a partiaUy received file. Instead, the clients store missed frames) ^ ^ nt ^ or rcmnds Each sub . 

partially received files. If there are problems that prevent all sequent pass or round requires tne transmission of fewer 

clients (eg., all clients in a multicast group) from receiving frames bccause on i y frames negatively acknowledged (i.e., 

the entire file when it is first being sent (e.g., the link is is frames not received or received in error) in the previous 

terminated for some reason during file transmission), the roun d get retransmitted in the subsequent round. 

transmission can be restarted later to complete the transfer. A maximum pass count or a maximum time to complete 

During a restart, the server queries all clients for a list of data can be a configurable parameter. There may be clients that 

frames that were missed, and then the server begins the have not received all of the file correctly by the time of a 

completion of the transfer by sending only those frames. 20 maximum pass or a maximum time duration. These clients 

Thus, in FIG. 1, for a restart, step 10 involves a transmission are identified by the server, and the server can take further 

that starts first with the frames that were missed (i.e., Nak'd) action to get these clients the rest of the information via, for 

during the initial aborted transmission, instead of starting example, a unicast file transfer process. In the preferred 

with the first frame of the first block of the file as would embodiment, the clients send "done messages" indicating 

happen in an unaborted normal start of a transfer. 25 they've received the whole file and the server sends "done 

Transfer lists" indicating chents said to be "done." If, after a prede- 

Upon entering the data transfer phase, a transmission log termined event (e.g., a predetermined amount of time), the 

is maintained at the server. This log is always on, and it server does not receive a "done message" from certain 

keeps track of all events. Each of the chents also maintains clients and all NAKs have been serviced, the server sends to 

a transmission log. The log maintained at each of the clients 30 those clients a status request message and sends any missing 

is mentioned hereinafter under the "COMPLETION" head- frames to clients needing more data. Any client that still is 

ing. unresponsive can be sent the file in, for example, a unicast 

As files having 2 gigabytes of data or more can be transfer to that client from the server at a later time, 

transferred, holding the entire file in memory at the server As the server passes block boundaries (i.e., B x , B 2 , B 3 and 

for the extent of the transfer generally is unrealistic. The 35 B 4 in FIG. 4), the individual clients preferably send "mul- 

number of clients which are to receive the file can be 1000 tiple selective reject negative" acknowledgments ("Nak") 

or more, and thus halting transmission to wait for acknowl- for each block. These acknowledgments from the clients for 

edgments from each of them before continuing on to the next each block are received at the server sometime after the 

block transfer is unacceptable. boundary of that block is passed. Positive acknowledge - 

The server logically breaks each file to be transferred into 40 ments are implicit. A multiple selective reject negative 

blocks of frames, and each block typically includes a plu- acknowledgment for a particular block means that one or 

rality of frames and possibly thousands of frames. Referring multiple frames in that particular block were received in 

to FIG. 4, in one example, the server 20 has broken a file into error, or were not received at all by those clients indicating 

four blocks, namely, Block 1, Block 2, Block 3, and, Block that the network did not deliver them for some reason. Thus, 

4, wherein each block includes one or more frames. Each 45 acknowledgments sent to the server indicate which frames 

block represents a unit that will be negatively acknowledged were received in error or not received, 

(only, no positive acknowledgements) by every client par- On subsequent passes (i.e., after the first pass shown in 

ticipating in a transfer when the client determines that a FIG, 4), clients only respond with negative acknowledg- 

block has been sent by the server. The client detects this by ments for blocks again not received correctly. Since the 

a change in block number in data packets received, because 50 server sends pieces (frames) of the file needed by various 

each frame sent indicates its block number and its frame clients to all clients in subsequent passes, many of the clients 

number within that block. Breaking the file into blocks will have already received it correctly on the first pass and 

provides at least two advantages: (i) decreasing the number thus will ignore it. 

of negative acknowledgments required; and (ii) reducing the In general, all information returning back to the server 

memory requirements in the server for determining next file 55 from the clients may be transmitted on a return path which 

pass transfer blocks. is separate from the path(s) which the server uses to transfer 

Data transfers are not directly tied to the negative the frames to the clients. However, for the purposes of this 
acknowledgments. Transfer continues regardless of missed description, the communications link 24 (FIG. 2), or other 
negative acknowledgments or previously missed data pack- path which allows the server and the clients to communicate, 
ets by any individual client. This allows simplicity of design 60 should generally be taken to mean both the server-to-client 
and ensures that individual client problems provide minimal fink and the return client-to-server link, 
impact on the group as a whole. Note also that clients are The server maintains various information about the trans- 
responsible for sending block negative acknowledgments fer and the participants in the transfer. In the preferred 
based on what they hear from the server. embodiment, this information is maintained by the server in 

Referring to FIG. 4, the server starts the transfer by 65 the form of data structures or lists. The server maintains and 

sending the first frame of the first block (i.e., the first frame uses this information to record and determine the status of 

of Blocki). The server sends the frames at a rate that is the file transfer. 
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The server also maintains a frame data structure which mission may be resumed or restarted at a later time, and, in 

indicates all selective rejects on individual frames from all the latter case, the clients will be requested to reinitialize 

clients. If multiple clients missed the same frame, the frame their contexts, 

data structure would indicate only that the frame was Completion 

missed. That is, the frame data structure is not maintained on 5 The server detects completion of individual clients by 

a client -by-client basis by the server. It generally is unde- receiving a "done message" from a client. The client knows 

sirable for the server to maintain a detailed list of missed it's done as soon as it has all blocks of the file, but the client 

frames on a per client basis because such a scheme would must continue to send "done messages" until the server 

use an inordinate amount of memory, particularly when a confirms completion. The server confirms that a client is 

large number (e.g., 1000 or more) of clients are involved in 10 "done" by placing that client's address in a "done list" and 

the multicast. For example, it might be that one or more of sending the list out to the clients. When a client sees its 

the clients either did not receive or received in error frames address listed in the "done list," it knows it has completed 

twenty and twenty-five of Blocks frame one of Blocks, the transfer. The client will then update its transmission log 

certain frames of Blocks etc. If the frame status maintained to indicate that the transfer has been successfully completed, 

by the server indicates that a particular frame of a particular is An ability to abort a transfer from the server or client is 

block needs to be retransmitted, it will be true that at least included. An abort packet provides the server and client with 

one of the clients has not acknowledged successful comple- the ability to abort prematurely a transfer. If the client sends 

tion of that particular block. After the server has sent the an abort, the server removes the client from the group. If the 

entire file once, the server would then pass through the frame server aborts, the transfer can be restarted without sending 

status information and reseod only the frames listed therein. 20 the full file on the first pass. 

This would continue, pass after pass, until all clients had sent Status Requests 

"done messages" and the frame status list is empty (or the If, after the first pass, the server has not received either a 

maximum number of rounds, or maximum time, had been DONE or NAK from a client, a query is sent directed to 

reached). those clients whose status is not known. The responses are 

Note that for any given pass, if any negative acknowl- 25 in the form of a standard response message. They will 

edgment does not get back to the server, the client will send include a bitmap describing the errors if there are errors to 

back to the server the same reject and retransmission request report 

messages during the next pass by the server. This means that Congestion/Flow Control 

if a certain client is not being heard by the server, that client As large "internets" become multicast enabled, it will 

will have to participate longer but that client will not 30 become more common to find multicast groups that desire 

appreciable impact the rest of the receiving clients. information to have different transmission links to the mem- 

Another piece of information stored at the server is bers of that group. These different links may have different 

statistics on the multicast group. When a transmission is capacities, which may be greatly divergent from each other, 

completed, summary information is provided on the trans- For example, one member of the group may have a link 

mission that can aid an operator in determining system 35 capacity of over 1 Mbps while another may only have 56 

performance problems and/or the performance problems of Kbps. In general, knowledge of these link capacities will not 

a particular client. be known by the sender (e.g., the server) of the transmission. 

Multiple Passes Through the File Thus, it is desirable to be able to determine the link 

Once the file has been completely processed once (i.e., capacities on the fly, and provide a flow control mechanism 

after the first pass or round), the transmission process 40 to prevent overload/congestion of the network while at the 

according to the invention will increment a pass counter and same time not inhibiting the efficiency of the data transfer 

then scan the frame status list in the server for the first block protocol. 

in which there was an error. Upon finding this first-error The data transfer protocol described herein includes the 
block, the server will resend the missed packets in that concept of blocks, each one of which can contain hundreds 
block. Negative acknowledgments for these missed packets 45 or thousands of frames. Clients (recipients) are obliged to 
will, as described previously, be generated by the clients send a multiple selective reject NAK at block boundaries if 
when they detect an error in a block. This is consistent with any frames are missing or in error in that block. For flow 
the first pass. All selective reject negative acknowledgments control purposes, it is desirable to gain knowledge of 
are indications of state and are therefore not specific to a missed/erroneous (i.e., dropped) frames as soon as practical, 
pass though they may change with each pass. In a preferred 50 so flow control decisions can be made. Changing or variable 
embodiment, the multiple selective reject negative acknowl- block sizes is a way to accomplish this with the data transfer 
edgments are in the form of bitmaps where the entire word protocol of the invention, and this involves starting with a 
represents a block and each bit in that word represents a relatively small block and increasing block size during file 
different one of the frames which make up that block. transfer to keep current scalability by reducing client 
Transmission Abort 55 acknowledgements. Another, preferred way to accomplish 
If, during the transmission, a fault is encountered which this with the data transfer protocol of the invention is to keep 
cannot be rectified, or if the operator manually aborts, a the block sizes all the same (homogeneous block sizes) but 
transmission abort sequence will be initiated. This sequence let the server send out status requests before a block bound- 
entails the repeated transmission of an Abort message for a ary occurs so that clients respond with NAKs before the 
certain interval (e.g., for an interval which is specified in a 60 block boundary. This latter technique is the most flexible, as 
transmissions file). The receivers acknowledge the abort NAKs may be solicited at any time, as opposed to just at 
message and can take action to, for example, either save the block boundaries which is the case with the former tech- 
context for a potential resumption (i.e., restart) of the nique. With either of these two techniques, NAKs are 
transfer or reinitialize the context to prepare for another solicited early in the transfer. 

transmission. There is a facility which allows the user to 65 In the "variable block size" method (the first technique 

initiate a transmission abort. A reason code can be set to mentioned in the preceding paragraph), the first block may 

either suspend or initialize. In the former case, the trans- be relatively small, e.g., 100 frames. Subsequent blocks 
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increase by a factor of two each time. Block sizes are the whole group can adjust without undue performance 

increasingly doubled until the maximum block size is degradation. The server's transfer rate for the group is then 

reached or the file reaches its end. dropped by 15%, or a higher or slightly higher percentage, 

In the "status request" method (the second technique accommodate client B. 

mentioned previously), the server solicits NAK responses at 5 The timing of the variable block size method is given in 

points where it desires, and these points are not at block FIG - 8 * ^ soon as information at a client indicates its frame 

boundaries. With this preferred embodiment for congestion dro P s exceed the Group Threshold, that client must take one 

or Jlpw cont rol, status req uests are sent at increasingly of ^ three above-listed alternative actions so that the group 

loneer intervals ~~ — transmission is not adversely affected. The adjustment of the 

*u it, j * * • . * f * • * group's transfer rate is performed after the second block has 

With both methods, transmission rate or transfer rate is set 10 f * . ^- -.l *i_ u • • c «_i i i ^ c 

, ., , , *i_ n j * r been sent, starting with the beginning of block 3. Transfer 

as described herein. However, rather than a fixed transfer fate cfa are ^ lemented a f b i 0 ck boundaries to provide 

rate, the settable rale represents an upper bound for the ^ Qn a Mock ^ &Qm ^ bbck NAKg ^ file 

transfer rate. After the first block (with t he variable blo ck transfer then proceeds with the transfer of block 3, which is 

s ize me thod^ wtoa ft Status reo j m s f is rwYt &ad (wiiiTthe set to be twice ^ krge as block 2> just as 51ock 2 ^ twice 

slaTuTrequest method), NAKs are f*7lt tn thr. satyerjxy is as large as block x xhis is followed by block 4, which is 

clients thajJiav e_dropped frames and this is an indication o f as large as block 3, and so on until the maximum block 

t ongeslion by those c ljepts tp, tfirf§erver._ size is reached or the file reaches its end, whichever occurs 

* If there are NAKs, the fact that they are directly related to first. However, if NAKs from the group after block 3 

the instantaneous capacity of the particular link can be used indicate the worst client exceeds a Rate Threshold parameter 

to determine link capacity for all of the links that show 20 (configurable, then the rate is further adjusted for block 5 

congestion based on the following equation: transmission. The Rate Threshold is the minimum frame 

drop percentage for which transfer rate adjustments for the 

((#frames sent-#frames NAK'd)/#£rames sent)*transfer rate-link group are performed. For example, a maximum frame drop 

capacity> percentage of 1% from the clients would not warrant an 

In the heterogeneous multicast network of FIG. 6, link 25 adjustment so the .Rate Threshold would typically be set to 

speeds range from 64 Kbps to 1024 Kbps, a large difference a number above 1%. 

in link capacity. Assuming noother traffic, if the transfer rate ' n slatus request method, the blocks are a uniform sue 

were set for 150 Kbps, the block NAK for block 1 from «^ s,a *f "f es * are xnt b * ^ .^f t0 r< ? U ^ NAKs 

client A (in the variable block size method) would indicate before block boundaries i are reached. Referring to FIG. 9, an 

about 58 frames dropped for the first block. Using the above 30 equivalent scenario to me one just described for the varying 

equation, the instantaneous link speed is calculated to be 63 «^e block method is depicted except now (in , FIG . 9) he 

Kbps. Tne block NAK from client B for block 1 (in the block sizes are homogeneous. In one example me first status 

variable block size method) would indicate about 15 frames K *™\ * 86111 f er 100 frame f °f ! ra ° ^ ' he KamA t 

dropped for the first block. Again using the equation, the f*< , 20 ? more frames are sent, etc. Cheat NAKs are sent 

instantaneous link speed for the link to B is calculated to be 35 to «» ser ? r at ***** ^ Mine times as m me variable 

127.5 Kbps. With other traffic present, the number of frames bI ?* f e ^T'.^ f , added flexlblh, y 

dropped would be higher resulting in a smaller calculated stah f ^quest method that status requests are sent 

link speed -o » at an y tune desired ra ther than having to wait for a block 

A Group Threshold parameter may be set by the user. The boundary to receive NAKs as is the case with the variable 

Group Threshold is the limit, expressed in percent of 40 bl °* size method. . . 4 „ 

dropped frames, by a particular client that is allowed for Wltb vanab ? block ( s ™. m * hod °\ f^ s 

continuing participation in the multicast group. If the Group re *™ { metbod ' 11 ^ IS \ ml deslrable ^ '? delete 

Tnreshold is set to 25%, it means that any clients in the S rou P membe K rs and h ™& a &- Deleted . **** 

group that have a frame drop percentage higher than 25% ca ? be col !!p ted mt0 anothe f r W °P eraUn § f a 

will need to take action so that the rest of the group is not 45 low " ra e " ™* lowe , r . f nsfer rate be d < l f r " 

adversely affected. In the example of FIG. 6, client A with n J! ned b y "lculauon on link capaciUes performed by 

58% of the frames dropped would need to take action. chents wh ?. leave ,hc ™* V™P can ^ be set ^ 

Clients will have enough information to make that decision al . a ma,cnm g ***** rate md a new transfer can be 

because the transfer rate and Group Threshold parameters L „ , \ . * ^ , , , . , ^ . . . 

are transmitted to clients in the Announce message. Clients 50 variable block size and the status request meth- 

which detect that their frame drop rate exceeds the threshold ^ ™ lo ™* c - 

may take ore of the following actions: ^ i- i /*r\ 

J Multicast can be in two forms: application layer (AL) 

1. Leave the Group and request from the server to be put multicast where the ne twork still delivers data to the entire 
into a lower speed group, with the group speed speci- 55 broadcast group? and multicast IP wner e the network routes 
fled based on the measurement made at that client; traffic based Qn multicast routers and Mernet specification 

2. Leave the Group without requesting further delivery, RFC 1112 is implemented in the clients. 

meaning that this client misses this transmission; and i n both cases, multicast groups are set up under initiation 

3. Suppress NAKs until a Status Request message is of the server. The server sends notifications on a unicast 
received from the Server, allowing the rest of the group 60 basis to clients to inform them of membership in a particular 
to finish without being held up by excessive retrans- multicast group. These multicast groups can be set up and 
missions from a high frame loss client (the transfer rate dismantled rapidly, allowing for a dynamic configuration of 
for retransmissions to this set of chents could be Lower multicast groups. For example, a multicast group could be 
to reflect their lower capacities). set up to be only in place for the transmission of a particular 

In the example of FIG. 6, the next highest percentage of 65 file, after which time the group was dismantled, 

frame drops comes from client B at 15% which is under the With AL multicast, the network still delivers traffic on a 

Group Threshold. This number represents a factor to which broadcast basis, but clients not in the group discard the data 
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not destined for it. When the group is set up, security keys example, if the packet size were 256 bytes, then the most bits 

may also be disseminated so that clients outside the group a packet could contain would be 256(bytes/packet)*8 (bits/ 

cannot read the data even if it happened that the data was not Byte) =2048 (bits/packet) which means that the largest 

discarded at that node (note that this could also be deployed allowable block size would be a block having 2048 packets, 

with multicast IP). Also, with AL multicast, the IP address 5 Although receiving nodes 22 can be interfaced to an 

remains a global or network-based broadcast address. As Ethernet LAN at 10 Mbps, WAN links are often of much 

with broadcast, this address becomes mapped to a broadcast lower speeds than that. Thus, an explicit transmit data rate 

address in the link layer protocol, e.g. a broadcast SMDS is settable/configurable. 

address. A multicast header is selected for the group and Receiving nodes can each experience resource problems 

becomes the group differentiator. 10 either prior to or during a transmission. Receiving nodes are 

With multicast IP, the network is a router network where enabled to query their resources prior to a transmission and 

the routers support Class D multicast IP addresses and determine if they have the facilities to receive the data. If 

multicast routing. The clients support RFC 1112, "Host no t, then they should either reinitialize space which is 

Extensions for IP Multicasting". RFC 1112 provides for host dedicated for the transmission or should indicate that they 

notification of their presence to the nearest multicast router 15 cannot participate in the transmission and corrective mea- 

for the purpose of update of router tables. sure can be undertaken through different channels. A facility 

A functional description of the above-described invention cou ld be provided where the server can force the availability 

is provided below. of disk space remo tely to allow the transfer of the file to take 

Referring back to FIG. 2, which generally can represent place, 

any broadcast or multicast IP router-based network, a pur- 20 ^ receivers 2 2 must also be aware of what they are 

pose of the invention is to enable the simultaneous trans- Usteiling for m<ffll a datagram ^ received on a dedica t e d 

mission of small or large data files (e.g., files up to 2 channel, the node 22 must determine if it is being addressed, 

gigabytes or more in size) by a server 20 to up to 5000 or ^ ^ can arfse when ^ appHcatiori is being used by 

more receiving nodes 22 over a wide area network (WAN) more ^ one ^ smissioa server 20 . There must be a way 

connection 24. The invention also is able to work over local 25 of ^^teeing that a receiving node 22 is participating in 

area networks and other types of communications links, as exactly one Emission at a given time. By dedicating a 

described previously. The transmission medium 24 can be UDP port to a server 20 and also relating an encryption key 

any type which supports the TCP/IP protocol stack in the to that it ^ ensured mat a receivillg node employing 

preferred embodiment. Other protocol stacks could also a promiscuous mode tap on the network 24 wiU not have the 

serve as the communications environment for the invention. 30 ability to be able to ^ t the transmitted data . 

Multicast can be supported in two ways: AL multicast and Some in&mn ^ Qa is maintained on the Ums . 

multicast IP, as mentioned previously. mission ^ 2Q ^ ferabl & a ^ of all lhe 

Files to be ransferred to the clients can be loaded onto the M nodes ^ ^ fa refer£nce 

server 20 via tape (e.g the tape drive 66 of HG. 5) or, if the mformation preferab i y is avai i able to allow theinformation 

files are small enough by floppy (e.g., the floppy drive of 35 ider tQ c me ^ m ^ case of service f 

FIG. S) Also, files to be transferred can be loaded onto the w ^ ^ feraW wm be & tiansmission data . 

server 20 vxa FTP (File Transfer Protocol), or some other bage WQere ^ d essed data flle ^ maintained 

unicast transfer mechan^m from the source of the file over read fof transmission> ^ e transmission database ^tuns 

a LAN or other netwo*, for example^ The files generaUy ^ ^ data ^ ^ descri tive information of u 

can be in any format. The data file is then read in from the 40 t for exa 7Q b identifyi Ve content of the files 

tape or floppy mto a file system of the transmission server _ , . . , , , , 

20. Note that the server 20 must have sufficient space . Each transmission preferably has a completion status 

available to read in an uncompressed copy of the data file. mdl f ator record a lo S f ^ err ° rs ^countered during 

For both services, the data file also can be encrypted so that ^ fnsmi^ion. There preferably also * an event file with 

noneligible receivers cannot receive and use the data file. 45 ^ list of aU the nodes for which the transmission failed, who 

Each transmission file preferably is uniquely identified. to caU ' ^ wn ^ 11 taiIed - 

There preferably is an indication as to its content and time At anv P omt in dme dmm Z lhe transmission, an operator 

of generation. The input files to the process can be over 2 m able to interrogate the status of the transmission as it 

gigabytes in size, and the system can also handle files much applies to the server 20 and each of the receiving nodes 22. 

larger than 2 gigabytes. 50 Alerts are generated if there are problems communicating to 

The file can then be stored on the server 20 and prepared certain clients or other problems. If any intervention is 

for transmission. Data from previous transmissions will indicated, the operator is allowed to initiate the corrective 

need to be readily available on the server 20 for some period action. 

of time in case they need to be retransmitted. A mechanism For ongoing maintenance and management of the service, 

for accessing the data is provided such that the data can be 55 the operator is enabled to maintain the list of receivers, 

readily queued-up for retransmission. transmission groups, transmission file descriptors, transmis- 

For efficiency, the file is transmitted in blocks. The size of sion parameters, and transmissions database. A background 

a block is derived from the largest packet (or block size can process will maintain the environment and both age data and 

be selected by the user) which can be transferred over the delete it according to housekeeping parameters, if enabled 

communications path 24. Its derivation is based on the fact 60 DV an alerted operator. 

that the clients will need to indicate to the server which of Data transmission according to the invention has been 

the packets in a block they failed to receive. One way, and described above. Further aspects of the invention are 

generally the simplest way, to do this is to send a bitmap described hereinafter. These further aspects include: SET- 

indicating by a bit setting positionally which packets were TABLE TRANSMISSION RATE; MULTICAST GROUPS; 

not received. The size of the block therefore is approxi- 65 MULTICAST PING; MULTICAST NETWORK PROBE; 

mately the number of packets which can be acknowledged SPEED GROUPS; and NEGATIVE ACKNOWLEDGE- 

in a bitmap which itself can be contained in a packet. For MENT COLLECTION. 
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Settable Transmission Rate _ identified by a list of client IP addresses, one address for 

As mentioned previously, it is possible to set the data each client in the multicast group, 

transmission rate. The example given previously illustrated There are two options for multicast groups; dynamic and 

when a settable rate is useful. In that example, the receiving static. For dynamic multicast groups, when the transfer is 

nodes 22 are interfaced to an Ethernet LAN having ao 5 complete, the group dissolves. Dynamic multicast groups 

available bandwidth of 10 Mbps and the WAN links con- are formed with ^ ANNOUNCE messages using multicast 

necting the LAN to other networks are of much lower speeds g rou P Class D addresses. In contrast to dynamic multicast 

than 10 Mbps. In such a case, the data transmission rate e rou P s > ^ static multicast groups, all of the members of 

would be set, according to the invention, to match, for the S™P r ™™ m ™ b «* of the S™? whe * * e transfer is 

example, the speed of the slowest WAN link. 10 COm P lcte : St f^ C mulhc f ^ S are formed b * th * ^ 

y j .i- c on a unicast basis and/or by using a common Class D 

in accordance won me invention, lor any given me ^ress to set up configurations. 

transfer session, the data transmission rate can be set ahead Multicast Pine 

of time. More specifically, the maximum bit rate at which ^ « ping >, ^ my ^ Tcp/Ip is very useful in determining 

data is transmitted during the session is settable. In a connectivity between two points in a TCP/IP network (i.e., 

preferred embodiment, it is set by setting a parameter to an 15 m determining if two points are actually connected). In 

integer value that represents the bit rate in kilobits per TCP/IP, a ping packet is sent to the desired end point which 

second (Kbps). For example, if this rate parameter has the reverses the addresses and sends it back to the sender. The 

value 56, it corresponds to a maximum bit rate of 56 Kbps. roundtrip time delay is also measured, and this is a mea- 

The rate parameter can be set to any value that corresponds surement of the time it takes for the ping packet to travel 

to the available bandwidth of the link connecting the source 20 from the sender to the desired end point and then back to the 

to the destination or to a value representative of a rate less sender. 

than the available bandwidth. That is, if the available band- It is also desired to provide a multicast ping utility, where 

width is 1 Mbps, the rate parameter can be set to any value all the members of a multicast group respond to the ping 

between zero and one -thousand, where 1000 Kbps equals 1 packet or ping request. Clients or hosts that support rnulti- 

Mbps. This ability to explicitly set the transfer rate allows 25 cast IP (RFC1112) will respond to a ping request with a 

long (in time) file transfers to coexist with other applications Class D IP address as the destination address. However, in 

on the network without hogging all or substantially all of the known multicast implementations, the sender of the ping 

bandwidth of the network. request only displays the first response it receives to its ping 

Multicast Groups request. That is, known multicast ping techniques do not 

"Multicast" is defined hereinabove as the case when the 30 make a network connectivity measurement, 
server node 20 sends data (e.g., a file) to a subset of all of A "Multicast Ping" feature of the invention displays all 
the client nodes 22 connected to the network 24. It is also multicast responses to a ping request thereby providing the 
disclosed hereinabove that multicast transmission can be in network connectivity information from source to group 
two forms: "application layer (AL) multicast" and "multi- recipients, and the roundtrip time delay information for each 
cast IF'. AL multicast is used when the network does not 35 multicast group recipient. In a preferred embodiment, this 
support the Internet specification RFC1112 but does support feature uses the standard ping ICMP messages, 
broadcast. If multicast IP is supported by the network As an enhancement according to the invention, it is also 
according to RFC1112 and multicast IP routing, it is rec- possible to use the Announce/Registration facility described 
ommended over AL Multicast. Multicast IP is used when hereinbefore as another form of the "Multicast Ping" feature, 
members of the group must support multicast and routers in 40 With this enhancement, Announce/Registration ping mes- 
the router network must also support some kind of multicast sages determine connectivity to the application layer of the 
routing protocol (e.g., DVMRP, MOSPF, or PIM). Unlike group recipients and back to the sender and roundtrip time 
AL multicast, Multicast IP is a true multicast protocol where delay information or each group recipient, 
only members of the multicast group receive the transmitted The "Multicast Ping*' feature thus allows network con- 
data. 45 nectivity and roundtrip delays to be determined by the 

For each file transfer, a multicast group can be defined sender for members of a multicast group, 

during the ANNOUNCE/REGISTRATION aspect of data Multicast Network Probe 

transmission, as describe hereinbefore. As stated, the server Multicast (the sending of one to many, but not to all) data 

maintains various information about the file transfer and the networks are just now starting to be implemented. Multicast 

participants or group involved in the transfer. In the pre- 50 IP, in particular, is new in router networks and can provide 

ferred embodiment, this information is maintained by the the mechanism for creating multicast groups over networks 

server in the form of data structures or lists. The server of all kinds, e.g., frame relay, SMDS, LANs, satellite, 

maintains and uses this information to record and determine wireless. The Internet also has the "Mbone" (multicast 

the status of the file transfer during the DATA TRANSFER backbone), a part of the Internet that supports multicast IP. 

stage. The client status structure includes a list of the status 55 The Mbone was started in early 1992 and has grown so 

of the participants of the multicast group based on data from that at the beginning of 1995 over 1500 subnets of the 

the announce registrations that are received by the server. Internet were multicast enabled. To this point, the Mbone 

Multicast group management is the process of assigning has been used as an experimental network by Internet 

clients to multicast groups. The task of organizing and researchers who have tested PC and workstation based video 

manipulating the list of clients in each group is the respon- 60 conferencing and whiteboard multicast applications, as well 

sibility of the application program that initiates file transfer as Internet "radio" and other experimental applications, 

in the first instance. The application program generally Multicast IP routing on the Mbone was initially imple- 

provides ease-of-use features such as associating a name mented in workstations using the multicast routing protocol 

with a client IP address, assigning a name to a group, etc. DVMRP; however, parts of the Mbone have had their 

Group management is required only at the sending station, 65 routers upgraded so they are multicast enabled. It is antici- 

e.g., at the server. A multicast group is specified when a pated that within 5 to 6 years the Internet will be fully 

sending station wants to transmit a file. The group is multicast enabled using the routers in the Internet. 



07/11/2004, EAST Version: 1.4.1 





Test Results with 400- frame Test File 




s 


# Naks 


#Naks 


# Naks 


# Naks 


# Naks 


Speed Sent 


for A 


for B 


for C 


for D 


forE 


64 Kbps 


0 


0 


0 


0 


0 


128 Kbps 


200 


0 


0 


0 


0 


256 Kbps 


300 


200 


0 


0 


0 


10 512 Kbps 


350 


300 


200 


0 


0 


1024 Kbps 


375 


350 


300 


200 


0 



6,151,696 

17 18 

As more of the Internet becomes multicast enabled, it will 

be used for mainstream multicast applications rather than as TABLE 1 
an experimental research tool. As this occurs, tools will be 
needed to facilitate usage. 

One large difference between the Internet and a private 

network is that the Internet is a very heterogeneous network. 

It is a network of networks, and there are large differences 

in the different parts of the network operated by different 

organizations. In contrast, many private networks are set up 

to be relatively homogeneous, with much control by the 

operator of the private network as to the architecture of the Referring to Table 1, the first run at a speed of 64 Kbps 

network. results in no negative acknowledgements (i.e., NAKs or 

, . . , , . , , 15 Naks) for any of the group members because all links 

Since the endpoints in the multicast network are likely to suppQrt 64 Kbpg Qr gfeater 

be linked at different rates with different networks, and ^ ^ is at 128 K bps, twice that of the first. In 

congestion in the network will be different at different parts tn is second run, client A has 200 NAKs, meaning that half 

of the network, it is desirable to be able to gain knowledge the frames are lost. This means that the speed of client A is 

of the capacity of the attached links in the multicast group, 20 64 Kbps (i.e., ((400-200)/400)*128 Kbps-64 Kbps). Clients 

and to test performance at that capacity, A "Multicast B through E exhibit no frame loss in the second run, and thus 

Network Probe" feature of the invention is designed to be mc s P ecd of cacn of ^ose clients is at least 128 Kbps. 

able to probe the Mbone or other large heterogeneous In ^ tbird mn > the s P eed of transfer * 256 ^P 5 ' and 

™ ii* * *, 1 t m * a- _ _ clients A and B exhibit 300 and 50 lost frames, respectively, 

multicast network from the trathc source and measure the _ _ , . „ . , A 4 , , - ™ 

- ^ . j > . j ii'i ■ 1 * c . , nr 25 Thus, from this third run, client As speed is 64 Kbps (i.e., 

capacity of the individual links quickly from that traffic ((40 o_300)/400)*256 Kbps=64 Kbps) which confirms the 

source. measurement from the second run. Also, in the third run, 

tw ■ ♦ in,- * u ♦ n-.fi client B's speed is 128 Kbps (i.e., ((400-200)/400)*256 

Referring to FIG. 6, a heterogeneous multicast network ^oJST \ or * ^ u V \1 • 

/ iL tTjtl c ii_ t j a u w Kbps-128 Kbps). Chents C through E have no errors m this 

(eg., the Mbone porfon of the Internet) has a multicast 30 ^ ^ ^ , h each a , ^ M fagt as 256 

group with five members, A through E, where each member Kbns 

of the group is connected by a different capacity link, i.e., a [n ^ f<mrth ^ ^ speed of ^ 5n Kbps Cliem 

different rate link. Member A of the group is tied to the Aexhibits 350 l ost frames so measures (400-350)/400*512 

network with a 64 Kbps (kilobits per second) link, B with a K5ps or 64 which checks ^ifa the p re vious measure- 

128 Kbps fink, C with a 256 Kbps link, D with a 512 Kbps 35 ments> client B exhibits 300 lost frames which measures 

link, and E with a 1024 Kbps link. The nature of these link ((400-300)/400)*512 Kbps or 128 Kbps which also checks 

connections is unknown to the server (i.e., the traffic source) with previous runs. Client C exhibits 200 lost frames which 

because connections to the Internet can be at many different measures ((400-200)/400)*512 Kbps or 256 Kbps. 

speed links. In the fifth run, the speed of transfer is 1024 Kbps, Client 

40 A exhibits 375 lost frames which measures to ((400-375)/ 

It is desirable for the traffic source to know the charac- 400)* 1024 or 64 Kbps as before. Client B measures ((400- 

teristics of the links to destinations so that it can optimally 350)/400)*1024 or 128 Kbps, and client C measures ((400- 

determine how to perform the multicast transfer of infor- 300)/400)*1024 or 256 Kbps. Client D measures ((400- 

mation to the destinations. If the application is a video 200)/400)*1024 or 512 Kbps. CUent E has no drops which 

conference, it may be determined that the quality to A at 64 45 means that its speed is at least 1024 Kbps. 

Kbps may be unacceptable, but the rest could participate at Thus, for each of the five runs, the capacity of a given link 

128 Kbps. Similarly, if file transfer is the application, groups is given by the following equation: 

D and E could make up a group operating at 512 Kbps ((#frames MW ^ )/#&ames seDt y speed of transmission^ 

transfer rate, while groups A, B, and C could operate at 64 ^ capacity. 

Kbps without exceeding network capacity. ^ technique also will take into account the traffic 

In accordance with the invention, the mechanism to probe on the link - For sample, if a physical link is 256 Kbps and 

the network to determine remote link capacity is the system the ' e is 128 Kbps of traffic on the link when the test is 

j . , . *lju Afo » /n *4« performed, the measurement will come up with a capacity oi 

and protocol described herein. After Announce/Registration __, * , « 

^. £ . * . , „ . , 55 128 Kbps. the remaining capacity when the traffic is con- 
to a multicast group of members A through E is used as a s jdered 

means to determine connectivity (Lc., to determine which &1 fof ^ ]emenli these tests can also be ^ d to 

members are actually connected to the server) in accordance ^ ^ ^ of ^ ^ ^ ^ ^ ^ 

with, for example, the "Multicast Ping' feature of the speeds to each client. For example, in FIG. 6, the link speeds 

invention desenbed in the preceding section, a test suite of 60 may be and it ^ dcsircd to tcst (he Unks with 

small files are sent in sequence at different speeds to the relatively long test patterns to determine frame error rates, 

group members. For example, a 400-frame test file may be For exarn pie, a 100,000-frame test file could be sent at 64 

sent first at 64 Kbps, then 128 Kbps, then 256 Kbps, then Kbps to the group consisting of members A through E. The 

512 Kbps, and finally at 1024 Kbps. Client negative ra te of transmission and the NAKs are stored at the source, 

acknowledgements will be received and stored at the server 65 and the number of NAKs from each client gives a measure 

as shown in Table 1 below, assuming no other traffic on the of the quality (i.e., the frame error rate) of each link. It could 

links. be expected that A would have the worse quality as it is the 



07/11/2004, EAST Version: 1.4.1 



6,151,1 

19 

most heavily loaded link, and E would be best as it is the 
least loaded. However, other factors could cause other 
results. Similarly, speeds can be increased and overloaded 
links may be deleted from the group to more heavily stress 
the higher speed links. 5 

Thus, using the "Multicast Network Probe" feature of the 
invention, the capacity of individual links can be measured 
quickly by the server if the individual link capacities are 
unknown. Also, if the link speeds are known by the server, 
this feature of the invention can be used to determine the 10 
quality of each link (i.e., to determine the frame error rate of 
each link). 

In accordance with this feature of the invention, the 
connectivity of the members of the multicast group is first 
determined by going through an ANNOUNCE/ 15 
REGISTRATION phase described hereinbefore. That is, the 
initial step is to determine which members of the group are 
connected to the server. Once the connected members are 
known, the test file transfer can begin to determine link 
speed or quality by the server sending a test file to each 20 
member and recording the results (i.e, the number of nega- 
tive acknowledgements for each group member). 
Speed Groups 

With the knowledge of the capacity, speed, or bandwidth 
of each of the various links interfacing the server to the 25 
clients (made available by, for example, the "Multicast 
Network Probe" feature described in the preceding section), 
a list of these speeds can be stored by the server. The list can 
then be used to generate or define a plurality of client groups 
based on link speed. For example, there may be two speed 30 
groups where one includes a client connected to the server 
over a link (or effective link) having a maximum possible 
speed of 64 Kbps and where the other one includes a client 
connected to the server over a link (or effective link) having 
a maximum possible speed of 1024 Kbps, The second group 35 
is thus much faster than the first group. What speed group a 
particular recipient is in affects the transfer of data to that 
recipient. During the initial pass of data transfer according 
to the invention, each of the recipients in the second, faster 
speed group will be sent all of the frames by the server, and 40 
each of the recipients in the first, slower group will be sent 
only every sixteenth frame (1/16) sent to the second group. 
This means that after the first pass, the server has sent all 
frames to the second group recipients, but it has only sent 
one-sixteenth of the total number of frames to the first group. 45 
The remaining portion of the frames not yet sent to the first 
group (i.e., 15/16 of the frames) are then sent to the first 
group recipients on subsequent passes. The point being that 
once the server knows the capacity of each member of a 
group, the server can tailor the data transfer to take advan- 50 
tage of the higher capacity links and not slow down the 
transfer of data thereto. 
Negative Acknowledgement Collection 

As mentioned previously hereinbefore, the number of 
clients which can receive a file according to the invention 55 
can number in the thousands. Thus, the number of entries in 
the client status list maintained by the server can number in 
the thousands. File transfer according to the invention cart 
be made more scalable. For example, it can be scaled to send 
a file to millions of recipients/clients instead of thousands of 60 
recipients/clients. In a preferred embodiment, these clients 
or recipients are members of a multicast group of clients. 

The scaling feature is helpful to avoid a potential problem 
when the number of clients in the group become too large. 
The problem is when a large number of clients send back 65 
negative acknowledgements to the file sender (e.g., server) 
and effectively choke the sender with more negative 
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acknowledgements than it can handle in a reasonable period 
of time. This causes the performance of the sender to drop 
as it needs to spend a significant amount of time receiving 
and processing the negative acknowledgements and it can- 
not attend to its other duties. This also clogs the link back to 
the sender to become jammed with the traffic of these 
negative acknowledgements. 

The solution to the problem is "negative acknowledge- 
ment collection" which in turn allows the number of client 
recipients to be dramatically increased from thousands to 
millions without clogging at the file sender/server 20. With 
this collection feature, certain clients or other network nodes 
act as "replication points" and collect block negative 
acknowledgements from other clients. In a preferred 
embodiment, these replication points (RPs) are routers. 
Referring to FIG. 7, five RPs are indicated across the United 
States, and the lines emanating from each RP are represen- 
tative of the one or more clients connected to that RP. For 
example, RP 100 has 1200 clients thereunder, RP 102 has 
900 clients, RP 104 has 100 clients, RP 106 has 800 clients, 
and RP 108 has 500 clients. The server or source 20 is 
located at another place in the United States. RP 100 collects 
all of the block negative acknowledgements from the (e.g., 
1200) clients associated therewith or connected thereto. The 
other RPs 102, 104, 106, 108 do the same for their associated 
clients. For each RP, after it collects all block negative 
acknowledgements from all of its associated clients, that RP 
sends on to the server 20, or to another RP in the chain 
heading to the server 20, just one acknowledgement mes- 
sage. That one message includes all of the block negative 
acknowledgements from all of the clients associated with 
that RP. When the server 20 eventually receives these 
collected block negative acknowledgement messages from 
the RPs, it sends back out on the next pass all of the frames 
negatively acknowledged. The RPs are responsible for 
receiving those subsequent-pass frames and forwarding 
them to the appropriate clients or other RP in the chain 
which will then forward them to the appropriate clients or 
other RP in the chain, etc. 

Variations, modifications, and other implementations of 
what is described herein will occur to those of ordinary skill 
in the art without departing from the spirit and the scope of 
the invention as claimed. Accordingly, the invention is to be 
defined not by the preceding illustrative description but 
instead by the following claims. 

What is claimed is: 

1. A method for determining the capacities of communi- 
cation links connecting recipients to a source, comprising: 

(A) determining which recipients are connected to the 
source by the communication links; 

(B) transmitting frames of data from the source to the 
recipients determined in step (A) at a predetermined 
rate over the communication links; 

(C) receiving acknowledgments from the recipients deter- 
mined in step (A), the acknowledgments including 
indications of frames requiring retransmission; 

(D) storing, at the source, the acknowledgments and the 
predetermined rate; 

(E) repeating steps (B), (C), (D), and (E) for a different 
predetermined rate until steps (B), (C), (D), and (E) 
have been repeated a predetermined number of times; 
and 

(F) determining capacity of one or more of the commu- 
nication links from information stored at the source. 

2. The method of claim 1 wherein step (A) is performed 
by the source sending a ping request to the recipients over 
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the communication links and the source receiving responses - 
from the recipients. 

3. The method of claim 1 further comprising, after step 
(F): 

(G) transmitting frames of other data from the source to s 
at least one of the recipients which is connected to the 
source by one of the communication links determined 

in step (F) to have a first capacity; and 

(H) transmitting a subset of the frames of other data from 
the source to at least one other of the recipients which 10 
is connected to the source by one of the communication 
links determined in step (F) to have a second capacity 
where the first capacity is higher than the second 
capacity. 

4. A method for transmitting data, comprising: 

(a) partitioning the data into one or more blocks wherein 
each block comprises one or more packets and each 
packet comprises one or more bits, said partitioning 
comprising setting the number of packets per block to 
be approximately equal to the maximum number of 
available bits per packet; 

(b) transmitting all of the packets in one of the blocks to 
one or more recipients; 

(c) receiving, from one or more of the recipients, indica- 
tions of packets in the transmitted block that require ^ 
retransmission, each indication associated with a dif- 
ferent one of the recipients and comprising a bitmap 
contained within a packet wherein the bitmap com- 
prises a number of bits approximately equal to the 
maximum number of available bits in the packet and 3Q 
each bit of the bitmap represents a different one of the 
packets in the transmitted block; and 

(d) retransmitting only those packets requiring retrans- 
mission. 

5. The method of claim 4 further comprising, before step 35 
(a), the step of setting a maximum transmission rate to a 
value less than or equal to an available bandwidth of a 
communications link over which the data is transmitted. 

6. The method of claim 4 wherein step (b) comprises 
transmitting all of the packets in one of the blocks to at least 40 
one recipient which is connected to the source by a first 
communication link having a first capacity and transmitting 

a subset of all the packets in one of the blocks to at least one 
other recipient which is connected to the source by a second 
communication link having a second capacity where the first 45 
capacity is higher than the second capacity. 

1. The method of claim 4 wherein step (b) further com- 
prises transmitting through a replication point and wherein 
the method further comprises: 

receiving at the replication point from one or more of the 50 
recipients, indications of packets in the transmitted 
block that require retransmission, each indication asso- 
ciated with a different one of the recipients and com- 
prising a bitmap contained within a packet wherein the 
bitmap comprises a number of bits approximately equal 55 
to the maximum number of available bits in the packet 
and each bit of the bitmap represents a different one of 
the packets in the transmitted block; 
passing on the indications associated with each of a 
different one of the recipients from the replication point 60 
as an indication of packets in the transmitted block 
requiring retransmission for all of the recipients; and 
retransmitting only those frames which the replication 

point indicated require retransmission. 
8. The method of claim 7 wherein the steps are repeated 65 
until no packets require retransmission, and wherein the 
replication point comprises a router. 



9. The method of claim 4 wherein the method for trans- 
mitting data is also for determining the capacities of com- 
munications links connecting recipients to a source and the 
method further comprises: 

before step (a), determining which recipients are con- 
nected to the source by the communications links, each 
recipient being connected to the source by a different 
one of the communications links; 

in step (c) transmitting the packets at a predetermined 
rate; and 

after step (d), the method further comprises; 

(e) storing the indications of packets in the transmitted 
block that require retransmission and the predeter- 
mined rate at the source; 

(f) repeating steps (c), (d), (e), and (f) for a different 
predetermined rate until steps (c), (d), (e), and (f) 
have been repeated a predetermined number of 
times; and 

(g) determining capacity of one or more of the com- 
munication links from information stored at the 
source. 

10. The method of claim 9 wherein determining which 
recipients are connected to the source is performed by the 
source sending a ping request to the recipients over the 
communication links and the source receiving responses 
from the recipients that are connected to the source, and 
wherein the recipients are members of a multicast group. 

11. The method of claim 9 further comprising, after step 
(g): 

(h) transmitting packets of other data from the source to 
at least one of the recipients which is connected to the 
source by one of the communication links determined 
in step (g) to have a first capacity; and 

(i) transmitting a subset of the packets of other data from 
the source to at least one other of the recipients which 
is connected to the source by one of the communication 
links determined in step (g) to have a second capacity 
where the first capacity is higher than the second 
capacity. 

12. The method of claim 4 wherein the method for 
transmitting data is for transmitting data to determine the 
error rates of communication links connecting recipients to 
a source, and wherein the method further comprises: 

before step (a), determining which recipients are con- 
nected to the source by the communication links, each 
recipient being connected to the source by a different 
one of the communication links; 

in step (b), transmitting all of the packets in one of the 
blocks from the source to the recipients at a predeter- 
mined rate; and 

after step (d), storing the storing the indications of packets 
in the transmitted block that require retransmission and 
the predetermined rate at the source and determining 
frame error rate of one or more of the communication 
links from information stored at the source. 

13. The method of claim 12 wherein determining which 
recipients are connected to the source is performed by the 
source sending a ping request to the recipients over the 
communication links and the source receiving responses 
from the recipients that are connected to the source, and 
wherein the recipients are members of a multicast group. 

14. The method of claim 12 wherein step (a) further 
comprises: 

sending a ping request from the source over the network 
to all of the members of the multicast group; 

receiving at the source responses to the ping request from 
all of the members of the multicast group; and 
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determining, at the source, roundtrip delay for the ping 
request to travel to each of the members of the multicast 
group and back to the source. 

15. The method of claim 4 wherein step (c) is accom- 
plished during transmission. 5 

16. The method of claim 15 wherein step (d) further 
comprises, after all the packets have been transmitted, 
repeating steps (b), (c), and (d) for only those indications of 
packets in the transmitted block that require retransmission. 

17. The method of claim 16 wherein steps (b), (c), and (d) 10 
are repeated, as recited in step (d), until no packets require 
retransmission. 

18. The method of claim 17 wherein steps (b), (c), and (d) 
are repeated, as recited in step (d), until a predetermined 
amount of time has passed. 15 

19. The method of claim 4 wherein: 

before step (a), the method further comprises defining a 
multicast group of recipients to receive the data, the 
multicast group comprising a subset of all recipients; 
and 
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step (b) further comprises transmitting all of the packets 
in one of the blocks to the multicast group. 

20. The method of claim 19 wherein step (d) further 
comprises, after all the packets have been transmitted, 
repeating steps (b), (c), and (d) for only those packets in the 
transmitted block that require retransmission. 

21. The method of claim 20 wherein steps (b), (c), and (d) 
are repeated until no packets require retransmission. 

22. The method of claim 20 wherein steps (b), (c), and (d) 
are repeated until a predetermined amount of time has 
passed. 

23. The method of claim 20 further comprising, prior to 
defining a multicast group, sending a ping request to all 
recipients and receiving responses from the recipients that 
are connected to a source of the ping request in order to 
determine which recipients are available to be in the mul- 
ticast group. 

* * * * * 
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